<html>
<head><meta charset="utf-8"><title>MBE: iterator interface for TokenSource · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html">MBE: iterator interface for TokenSource</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="163086655"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163086655" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163086655">(Apr 11 2019 at 10:07)</a>:</h4>
<p><span class="user-mention" data-user-id="216201">@Edwin Cheng</span> split into the new stream</p>



<a name="163086755"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163086755" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163086755">(Apr 11 2019 at 10:08)</a>:</h4>
<p>SO yeah, I feel like making <code>TokenSource</code> an iterator-ish thing with explicit <code>bump</code> method would work</p>



<a name="163086776"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163086776" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163086776">(Apr 11 2019 at 10:09)</a>:</h4>
<p>The two probels we need to sovle are</p>
<ul>
<li>lookahead</li>
<li>composite tokens</li>
</ul>



<a name="163086866"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163086866" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163086866">(Apr 11 2019 at 10:10)</a>:</h4>
<p>I think we always do a constant amount of lookahead, and that lookahead should never look <em>inside</em> of a token tree, so</p>
<p><code>fn lookahead(&amp;self, n: usize /* n &lt;= 3*/) -&gt; SyntaxKind</code> would work</p>



<a name="163086906"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163086906" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163086906">(Apr 11 2019 at 10:11)</a>:</h4>
<p>For composite tokens, we never lookahread them, IIRC, so having an</p>
<p><code>fn at(&amp;self, composite: SyntaxKind)</code> and <code>fn bump_composite(&amp;self, kind: SyntaxKind)</code> should work</p>



<a name="163086975"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163086975" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163086975">(Apr 11 2019 at 10:12)</a>:</h4>
<p>And <code>fn lookahead</code> will not read the parent / child level, right ?</p>



<a name="163087020"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087020" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087020">(Apr 11 2019 at 10:13)</a>:</h4>
<p>right, lookahead  is stricttly within one level</p>



<a name="163087226"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087226" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087226">(Apr 11 2019 at 10:16)</a>:</h4>
<p>If so, i think it can be simplify a lot of current implementation of tree-traversal.</p>



<a name="163087247"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087247" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087247">(Apr 11 2019 at 10:16)</a>:</h4>
<p>yeah, that's the hope!</p>



<a name="163087354"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087354" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087354">(Apr 11 2019 at 10:18)</a>:</h4>
<p>Just reiterate : </p>
<div class="codehilite"><pre><span></span>trait TokenSource {
   fn current(&amp;self) -&gt; SyntaxKind;   // Can composite
   fn is_at_composite(&amp;self, kind: SyntaxKind);
   fn bump(&amp;mut self); // Can not composite
   fn bump_composite(&amp;mut sekf, SyntaxKind); // Can composite
   fn lookahead(&amp;self, n: usize /* n &lt;= 3*/) -&gt; SyntaxKind; // Can composite ??!
}
</pre></div>


<p>right</p>



<a name="163087364"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087364" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087364">(Apr 11 2019 at 10:18)</a>:</h4>
<p>?</p>



<a name="163087419"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087419" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087419">(Apr 11 2019 at 10:19)</a>:</h4>
<p>I guess, both <code>current</code> and <code>lookahead</code> should <strong>not</strong> composite</p>



<a name="163087480"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087480" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087480">(Apr 11 2019 at 10:20)</a>:</h4>
<p>that is, one should just call <code>is_at_composite</code> explicitelly instead</p>



<a name="163087513"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087513" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087513">(Apr 11 2019 at 10:20)</a>:</h4>
<p>so, if parser is at <code>&lt;&lt;</code>, current returns <code>&lt;</code>, lookahead on 0 and 1 returns <code>&lt;</code>, but is_at_composite('&lt;&lt;') returns true</p>



<a name="163087543"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087543" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087543">(Apr 11 2019 at 10:21)</a>:</h4>
<p>That's great !!!</p>



<a name="163087670"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087670" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087670">(Apr 11 2019 at 10:23)</a>:</h4>
<p>If so,  we maybe even do not need ttToken struct, we could compute the value on the fly.</p>



<a name="163087737"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087737" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087737">(Apr 11 2019 at 10:24)</a>:</h4>
<p>yeah, I guess that is true!</p>



<a name="163087743"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087743" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087743">(Apr 11 2019 at 10:24)</a>:</h4>
<p>Okay , i think the remain problem is TreeSink trait ?</p>



<a name="163087867"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163087867" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163087867">(Apr 11 2019 at 10:26)</a>:</h4>
<p>why is it problematic?</p>



<a name="163088001"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163088001" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163088001">(Apr 11 2019 at 10:29)</a>:</h4>
<p>the current TreeSink::token is defined as following: </p>
<div class="codehilite"><pre><span></span>fn token::(&amp;mut self, kind: SyntaxKind, n_tokens: u8)
</pre></div>


<p>So we need to roll back the cursor and traversal it again to collect the tokens, do you think there is a better solution ?</p>



<a name="163088104"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163088104" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163088104">(Apr 11 2019 at 10:30)</a>:</h4>
<p>Hm, I wonder why we need n_tokens there at all</p>



<a name="163088108"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163088108" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163088108">(Apr 11 2019 at 10:30)</a>:</h4>
<p>like SyntaxKind should determine <code>n_tokens</code> exactly?</p>



<a name="163088188"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163088188" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163088188">(Apr 11 2019 at 10:32)</a>:</h4>
<p>I gues, if <code>TokenSource</code> is a mutable iterator, for MBE token-sink can be 100% no-op?</p>



<a name="163088195"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163088195" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163088195">(Apr 11 2019 at 10:32)</a>:</h4>
<p>like we just see how far the iterator is advanced</p>



<a name="163088339"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163088339" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163088339">(Apr 11 2019 at 10:35)</a>:</h4>
<p>Why do we need treesink of TokenSource is mutable iterator ?</p>



<a name="163088343"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163088343" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163088343">(Apr 11 2019 at 10:35)</a>:</h4>
<p>Why do we need treesink if TokenSource is mutable iterator ?</p>



<a name="163088345"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163088345" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163088345">(Apr 11 2019 at 10:35)</a>:</h4>
<p>for macros, we probably don't need it at all</p>



<a name="163088361"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163088361" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163088361">(Apr 11 2019 at 10:35)</a>:</h4>
<p>but for actual parser, where we have a flat list of tokens and want a syntax tree in the end, we do need tree sink</p>



<a name="163088537"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163088537" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163088537">(Apr 11 2019 at 10:38)</a>:</h4>
<p>Okay, make sense.</p>



<a name="163088621"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%3A%20iterator%20interface%20for%20TokenSource/near/163088621" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.3A.20iterator.20interface.20for.20TokenSource.html#163088621">(Apr 11 2019 at 10:40)</a>:</h4>
<p>Thats enough for me to try implement a PR for this change,  after i finish the add LR_DOLLOR PR :)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>